The present invention relates generally to electronic data storage management, and more particularly, to systems and methods for electronic data storage management on multiple device types and multiple devices.
Information is rapidly becoming the most valuable asset of most companies. At the same time, managing and protecting information is becoming dramatically more complex and difficult due to the explosion in data storage requirements and the shift from centralized to distributed storage of data on enterprise networks. In addition, users are storing more and more data on their desktop and laptop computers. Effectively managing and protecting this critical information is left to information technology (IT) professionals and represents one of their greatest challenges, perhaps even rivaling the Year 2000 challenge.
The financial impact of information loss is enormous. Not only is it costly to recreate lost data but also there are other significant costs such as lost productivity and lost opportunity.
Over the years, IT departments have evolved highly sophisticated and effective means of managing centralized mainframe data, including backup, disaster recovery, and hierarchical storage management (HSM). Achieving the same level of effectiveness and protection at a single workstation, or in a distributed network environment, however, presents quite a challenge.
Since the early days of computing, there has been recognized the need to protect and manage the data that is stored on computers for disaster recovery and archival purposes. Historically, the standard storage management solution was tape backup. With tape backup, data on computers disk memory storage is periodically backed-up to tape. These backups were generally done during off business hours with cycles of full backups and then intermediate incremental backups of files that had changed since the previous backup.
Numerous problems existed with tape backup:
If any data was entered and lost during the time between the scheduled, periodic backups, the data would not be recoverable except by re-entry;
As computers proliferated in organizations, performing tape backups on multiple machines became increasingly time-consuming and difficult to manage. To help overcome this problem, network access to remote machines was added by many backup software vendors;
Backups generally needed to be done during off-business hours due to the negative effect on computer performance and also because many files are locked by applications;
Restoring a file or system from tape backups is generally very time consuming.
Historically, if more free disk space was required on the computer, the operator could create an archive of selected data on tape. After the data was moved to tape, the data could be removed from the hard drive. One problem with this method of migrating data to tape is that if the end user needed to later access the archived data, there was no indication that the file had been migrated. Another problem was in selecting which files would be most suitable for migrating to tape.
Hierarchical Storage Management (HSM) was created to automate the process of data migration. With HSM, files in primary storage are selected for migration to secondary storage according to some criteria, for instance, the file least recently accessed could be selected for migration. With HSM, when the file is moved, a marker file is put in it""s place. Depending on the implementation, if the user attempts to access the marker file, the file is returned to its original location or the user is prompted to restore the file from a given tape or removable cartridge. One common problem with this technique is that the file must be restored to its original location. If, during a restoration attempt, HSM determined that adequate space was not available to restore the file to its original location, then the file cannot be accessed.
Extension of disk space across local physical devices can be found built into some file systems such as Window NT""s NTFS. For example, in Windows NT, a single disk could be extended by creating a volume set which creates a logical drive that may extend across multiple physical volumes. (Hereinafter, a xe2x80x9cvolumexe2x80x9d refers to a single storage unit. Reference to a physical volume refers to a single physical storage unit, such as a single disk, a single CD ROM, a single tape, etc.; reference to a logical volume or xe2x80x9cvirtualxe2x80x9d volume, refers to data that is located on one or more physical storage units but is treated by the computer system as if the data is located on a single physical storage unit. Reference to a xe2x80x9cdevicexe2x80x9d refers to a piece of electronic storage equipment that is capable of accessing a particular type of storage volume, such as a disk drive, a CD ROM drive, a tape drive, etc.) While this accomplished the goal of virtually extending the logical drive, this method is limited to only certain file system formats (the method applies to, for example, NTFS, but not to Windows FAT 16 or FAT 32). Also, this technique lacks flexibility in that space cannot be easily shifted from one volume to another.
Another technique for creating an expanded volume was the use of a Distributed File System such as Microsoft""s DFS. Using DFS software, a logical volume can be created where each subfolder may point to a folder on a different volume on any machine in the network. This approach is very useful for creating logical directory structures independent of volume location. While DFS can be used for extending disk space, it was not designed as a disk space extension tool and has a number of limitations. First DFS cannot be used for extending an existing local volume. Further, if a particular volume in the DFS tree runs out of space there is no provision for extending space in that folder across another volume.
Storage is growing at an exponential rate. International Data Corporation (IDC) has forecasted a 90 percent annual increase in total storage shipments for the next four years, with annual storage shipments surpassing 500,000 terabytes by 2001. That""s more than all the data storage shipped during the 1990s. IDC also estimates that the average site capacity will double annually.
The explosion in storage requirements has created two major problems for IT professionals:
Storage Management
It is becoming increasingly difficult to manage and protect the ever increasing and more widely distributed amount of data being stored. As a result, more critical data is being exposed to loss.
Storage Capacity
The exploding storage requirements are continually outgrowing the storage capacity of servers and workstations. As a result, storage capacity must be continually updated, which is costly and disruptive to users.
According to Bear Stearns, more than 50 percent of critical data is now stored on desktop and laptop computers. This data is outside the reach of most enterprise storage management software products and is at risk to loss. To get around the problem, many IT departments have encouraged their users to store critical data on network servers to bring it within the reach and control of storage management software. This policy, however, has met with only limited success. Laptop computers present a particularly difficult storage management problem in that most are only occasionally connected to a network and must be able to work offline. This makes storing active data on servers impractical.
There are many ways that users can corrupt or lose information, including:
Inadvertent File Deletion
A user may unintentionally delete a file that contains critical information. Some protection solutions address this problem, such as the Windows Recycle Bin and utilities that hold information deleted from the Recycle Bin. Even with these solutions, however, users can still lose important data.
Inadvertent File Overwrites
A user may wish to maintain each revision of a document but unintentionally save a new revision of a document over the old one.
New Software Install
In installing or updating software, a user may xe2x80x9cbreakxe2x80x9d a working system. That is, the installation process may corrupt or destroy critical information such as system files. This could result in the system no longer working properly.
Lost or Stolen Computer
This problem is primarily associated with laptop computers. When a computer is lost or stolen, all its local storage is lost.
Hardware Failure
This has been a longstanding problem and is even more prevalent in portable computers that are subject to rough treatment.
Attack
Software viruses, Internet hackers, and disgruntled former employees all represent a threat to computer data.
Natural Disaster
Fire, flood, tornado, hurricane, and earthquake can result in significant data loss.
The loss of productivity and disruption caused by information loss can be enormous. Even more significant than the cost of recovering or recreating the lost information is the negative impact on user productivity. Consider, for example, the impact of a sales representative losing a laptop computer containing all customer and prospect contact information.
According to Peter Teige, a Senior Research Associate with Mobile Insights, xe2x80x9c[i]n our research we found estimates that as many as 25 percent of mobile PC users experience some major data loss each year due to system failure, loss, or theft. When you break it down, this loss of data translates into significant financial losses. There are estimates that each unrecoverable data loss from a mobile PC represents a financial loss of more than $60,000, on average. It really begins to add up if you take into account all the associated costs, such as lost productivity, investment, opportunity cost, and even the cost of employee turnover that results.xe2x80x9d
The plummeting costs and increasing capacities of storage devices make it economically feasible to provide protection through replication of distributed data in centralized storage facilities. Hierarchical storage management systems combined with large-scale tape and optical disk hardware, plus storage robots, allow organizations to optimize storage utilization and further reduce the cost of network ownership.
Organizations need a storage management solution that enables them to cope with the rapidly increasing storage requirements, the shift from centralized to distributed storage, and the increasing proliferation of data on desktop and laptop computers. To be effective, a storage management solution must meet a number of requirements.
There are three primary types of stored information, all of which should be protected by an effective storage management solution:
Recoverable Information
Information such as operating system or application software that can be recovered through reinstallation if lost.
Unrecoverable Information
Information such as data, documents, presentations, or spreadsheets that cannot be easily recreated, if they can be recreated at all.
Temporary Information
Information, such as scratch files, that is created and used by applications only while they are running and typically deleted when the application is closed.
The effective storage management solution should address recoverable as well as unrecoverable information. The need to protect unrecoverable data is obviousxe2x80x94unless it is protected, it must be manually recreated if it is lost.
The effective storage management solution should also protect recoverable data. Although this information may be recoverable through a reinstall process, the reinstall process can be extremely time consuming and typically requires user input and interaction. For example, in reinstalling an updated application, a user must install the application plus all the updates. In addition, the user must reset all favorites, preferences, and options to restore the application to its exact state prior to the loss.
It is important that data be protected on a real-time continuous basis as opposed to a scheduled periodic basis. That means, every time a file is changed, the changes should be immediately recorded. Continuous protection provides more coverage than scheduled periodic protection in that a user can activate backup by simply saving a file after making changes. In this way, the amount of work lost in the event of a system or application failure is minimized.
An optimal storage management solution should provide for versioning of data, application programs and the operating system. In many cases, a user needs to return to an earlier operating version of a system if a problem occurs during the installation or update of an application or system software. For example, if an application update breaks the application so that it no longer works, or worse yet, breaks the operating system, then the user should be able to return to the previously working version of the application or system. In an optimal storage management solution, the user should be able to rewind to the previously working version of the application or system rather than reinstalling the original application plus all its updates and resetting all preferences and options.
An optimal storage management solution not noticeably impact the performance of desktop or laptop PCs. Wherever possible, storage management software should operate in the background and allow users to continue with their jobs with little or no interruption or delay.
The present invention meets the requirements discussed in the previous section. The present invention applies to implementation on an individual computer workstation and to implementation on a local area network (LAN) environment.